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ABSTRACT 



This guide is designed to help districts and schools, 
participating in the federally sponsored Comprehensive School Reform 
Demonstration Program, meet the evaluation requirements of the program as 
well as their own needs for useful evaluation information. Much of the 
information is applicable to any local jurisdiction engaged in school reform 
that wants to know how its initiatives are working and the changes needed to 
improve them. The guide presents a five-stage evaluation process which 
organizes the array of evaluation activities into clusters that are 
consistent in timing and purpose. The five stages are: (1) planning the 

evaluation; (2) designing the evaluation; (3) conducting the evaluation; (4) 
reporting the findings; and (5) encouraging use of the findings. Each chapter 
focuses on one stage of the evaluation. The appendix contains blank 
worksheets for use in the evaluation process. (Contains 2 figures, 3 tables, 

8 exhibits, and 18 references.) (SLD) 
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INTRODUCTION 









F or many educators, talking about comprehensive school reform brings 
to mind Yogi Berra’s famous saying, “It’s deja vu all over again.” Catch 
phrases like “continuous school improvement,” “effective schools,” and 
“whole school reform” have come and gone. Indeed, they seem to “erupt 
noisily every decade or so, often only to recede quietly into the background” 
(Murphy, 1990). 

In some ways, the notion of comprehensive school reform is not new; at least 
the elements that make up what’s called comprehensive reform are not new. 
However, comprehensive school reform attempts to move beyond previous 
reform efforts by bringing together much of what has been learned so far 
about creating better schools. The accountability of the education system for 
realizing the improved performance of all students is a driving force behind 
these most recent initiatives. 



In the past, efforts to improve public education have focused on fixing the 
parts of schools, the people in the schools, or the schools themselves. Through 
these efforts, researchers and practitioners have come to understand that in 
order to improve student achievement, it is necessary to rethink and revamp 
the whole education system — from the classroom to the district office to the 
statehouse. This top-to-bottom overhaul is the essence of comprehensive 
school reform. This approach goes beyond the piecemeal improvement efforts 
of the past, not just by combining them, but by integrating them 1 (Sashkin 
& Egermeier, 1993). 

Although the pieces of comprehensive school reform may not be new, 
understanding how to put them together in a way that changes schools for the 
better is new. Like any other pursuit, education is evolving. To ensure that we 
learn from our experiences and that continuous improvement efforts move 
forward, it is essential that evaluation be an integral component of reform 
initiatives. Evaluation should not just assess the outcomes and impact of new 
strategies; it also should guide the process of implementing and refining 
reform programs. 

1 The movement toward a schoolwide strategy under Title 1 is an example of this integrated 
approach. 
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Down the road, there certainly will be new directions and innovations in 
education. But that will not mean that current methods for creating better 
schools will have been proven wrong. Rather, it will mean that educators have 
acquired new knowledge about how to make schools successful. More 
important, it will mean that researchers and educators have not tired of trying 
to make this nation’s schools the best they can be. 

Purpose of This Guide 

This guide is designed to help districts and schools participating in the 
federally sponsored Comprehensive School Reform Demonstration (CSRD) 
Program meet the evaluation requirements of the program, as well as their 
own needs for useful evaluative information. However, much of the 
information provided in this guide is applicable to any local jurisdiction 
engaged in school reform that wants to know how its initiatives are working 
and the changes that are needed to improve them. Thus, schools that intend 
to implement comprehensive school reforms using other funding sources 
should also find this guide useful. 

This guide is based on the assumption that your district or school has already 
decided to implement a comprehensive school reform initiative and has 
identified a model or strategy . 2 We also assume that a plan is in place, 
funding is available, and that you are now facing a challenge related to 
evaluation. Almost certainly, this new program is adding a layer of data 
collection and reporting on an already overburdened staff — merely reporting 
student test scores no longer is sufficient; you are now expected to participate 
in evaluation planning, data collection, and data analysis. 

The good news for those participating in the CSRD Program is that the 
program is intended to serve as a “reform umbrella” for other reform efforts 
underway; more specifically, the evaluation and data reporting requirements 
of other efforts can be integrated with CSRD requirements. In addition, 
generating the information you need to make improvements is under your 
control. Therefore, the new evaluation challenges you are facing present a 
unique opportunity for informed decision making at the local level. After 
reading this guide, you should be better able to 

• plan a focused, practical school reform evaluation; 

• develop a detailed evaluation plan; 

• conduct an evaluation of school reform implementation and impact; 

• report key findings in formats useful to major audiences; and 

• encourage use of evaluation findings for program improvement. 

2 Should assistance be needed in selecting a reform model, Making Good Chokes (Hassel, 1998) 
is an excellent resource available from the North Central Regional Educational Laboratory. 
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This guide does not address the various philosophical underpinnings of 
evaluation theory and practice, compare and contrast evaluation models, or 
quibble over the fine points of defining the meaning of evaluation. Rather, it 
provides practical information, tips, and tools, which together serve as a 
resource for novice and expert evaluators alike. 



Using This Guide 

Evaluation is not a “one size fits all” proposition. We recognize that the 
specific details of each locally designed evaluation of school reform will vary 
depending on the program crafted and the goals and benchmarks established 
to assess progress. This guide presents a five-stage evaluation process to 
organize the array of evaluation activities into clusters that are consistent in 
timing and purpose. The five stages are 

1. planning the evaluation, 

2. designing the evaluation, 

3. conducting the evaluation, 

4. reporting the findings, and 

5. encouraging use of the findings. 

In the Appendix of this guide, you will find blank worksheets. These worksheets 
are for your evaluation team to copy and use during the evaluation process. 
Throughout this guide, you also will find exhibits. These exhibits are not 
intended to be exemplars of what a particular district or school should try to 
create; rather, they are included to illustrate a range of situations, responses, and 
data options that might emerge when completing the worksheets provided. 

To assist more experienced evaluators who may be interested only in 
completing the series of worksheets provided, special headings in purple 
italics are provided to highlight the sections of the guide that directly relate 
to completing worksheets. These sections also are listed in the Table of 
Contents (e.g., “Using Worksheet 1 to Align Requirements”). In addition, 
shaded boxes are used throughout this guide to highlight “evaluation tips” 
that are discussed in more detail in the main text. 

Finally, each chapter focuses on one stage of evaluation. The stages are 
presented in the sequence in which they are typically conducted. Readers who 
are just beginning their evaluation may find it most useful to read through the 
chapters of the guide in the sequence provided. Readers who have already begun 
their evaluation may choose to use the guide as a resource and turn to the 
chapter that best matches the evaluation stage they are currently involved in, 
returning to earlier chapters only as needed. Regardless of the stage of 
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evaluation that your school or district is involved in, as the evaluation 
progresses, you may want to revisit chapters to help you modify your original 
design, data collection methods, or analysis plan. 



CSRO Program Keqylremerali’s 

The Comprehensive School Reform Demonstration (CSRD) Program was 
implemented as part of the FY 1998 Labor-HHS-Education Appropriations 
Act passed in November 1997. The program was created by Congress to help 
schools substantially improve student achievement. The financial incentives 
provided under this federal initiative signal a shift away from support for 
more limited or targeted interventions to support for more sweeping, 
comprehensive school reform strategies. The multi-year initiative, funded at 
$145 million in each of its first two years, seeks to involve more than 2,500 
schools across the nation in implementing research-based comprehensive 
school reform programs to help see that all children meet challenging content 
and performance standards. 

In an effort to ensure that only comprehensive school reform programs are 
funded under this initiative, the legislation clearly stipulates the 
characteristics of qualifying programs and goes so far as to list examples of 
established reform models that might form the basis of local comprehensive 
reform programs. 3 (However, the legislation does provide for the 
implementation of locally developed models.) 

Perhaps the most important feature of this new federal initiative is its focus 
on coherent, schoolwide reform efforts, rather than on a collection of 
independent, uncoordinated reform projects. This focus encourages schools to 
align existing funds, say from Title 1 programs, with CSRD funds to increase 
the prospects for successful comprehensive school reform. 

Those involved in planning and conducting an evaluation of local reform 
efforts launched under this federal initiative should consider the nine program 
criteria shown in Table 1, the characteristics of the specific reform model 
selected or created, state department of education evaluation requirements, 
and the local context in which the effort will be undertaken. An evaluation 
plan (criterion 8) is itself one of the federal program requirements for 
participating schools; this plan should be built around the measurable goals 
and benchmarks (criterion 4) established for a CSRD Program. 



3 A review of these and other comprehensive school reform models is provided in the Catalog 
of School Reform Models (1st ed.) and its Addendum (August 1999). These materials are 
available online at http://nwrel.org/scpd/natspec/catalog. 
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CSiR© Program CG-ifwii® 
so$ OderDfiffied] ora tfShe IFedeireifl (LegosOsafro^tni 

1. Employ effective, research-based methods and strategies. 

2. Have a comprehensive school reform plan with aligned 
components. 

3. Provide high-quality professional development. 

4. Have measurable goals and benchmarks. 

5. Have support within the school. 

6. Provide meaningful parental and community involvement. 

7. Use high-quality external technical support and assistance. 

8. Have an evaluation plan. 

9. Use other resources to coordinate services and support and 
sustain reform. 



Wheat Y@y Slh©u0d] §Cirs@w EvdSy®ffi®ini 

There are many, defensible ways to carry out educational program evaluations. 
Often the best approach is eclectic, combining what appears to be best from 
various methods rather than adhering to a particular evaluation model. The 
visibility of a program evaluation can be increased by considering it an 
integral component of the overall reform effort. The evaluation should provide 
information to all stakeholders about progress toward stated program goals 
and about the impact and outcomes realized. Because many outcomes take 
considerable time to emerge, it is important to maintain a continuous flow of 
evaluative information, making corrections and adjustments along the way to 
increase the likelihood that outcomes will be realized. 

Examine process and impact. A program evaluation certainly should 
document the intermediate and long-term outcomes of a reform effort. A 
truly useful evaluation also should examine the adequacy of implementation 
processes in view of the outcomes observed. In short, an evaluation should 
provide feedback to decision makers and participants about needed program 
modifications and improvements. Evaluation should be an integral and 
ongoing part of the reform program; it should be embedded in program 
operations rather than tacked on as an additional administrative task to meet 
someone else’s requirements. Thus, evaluation planning is best begun in the 
early stages of the program “start-up” phase. 
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Create a flexible design. Just as program adjustments should be made when 
necessary, so, too, should evaluation designs be flexible enough to allow for 
modifications when needed. In this way the evaluation can respond to 
important feedback and insights about unanticipated problem areas. If a 
program element does not seem to be working well, it must be possible to 
adjust the evaluation plan to obtain the needed evaluative information. For 
example, if teacher training is not going well, the evaluation could be altered 
to include a focus group of teachers to assess the nature of the problem. 

Keep design realistic . The evaluation effort should be as comprehensive and 
rigorous as possible, yet realistic in scope and demand given available 
resources. There are many aspects of a complex program that can be evaluated. 
Your job is to identify the most important questions and give them priority. 
Data collection can be expensive, but time and money are always limited; 
therefore, resources should be directed to learning about the most critical 
program components and issues. One strategy for using scarce resources 
wisely is to be clear about why particular information is being collected. It is 
useful to know — in advance — how data sets will be used to answer specific 
questions. If there is no clear reason to collect a data set other than it is 
available or relatively easy to obtain, it should be eliminated from the 
evaluation design. A carefully chosen evaluation consultant, brought in at the 
design stage, can help you identify cost-effective ways to focus the evaluation, 
maintaining acceptable standards while maximizing available resources. 

Establish an evaluation team. An evaluation team is essential to the 
successful design and ongoing maintenance of an evaluation effort. Each 
member must be given release time to focus on these new duties. A small, 
core team of staff members should be established, and opportunities for 
discussion and coordination throughout the building should be frequent and 
planned. One person should be designated as the point of contact for 
evaluation-related issues and questions; this team leader is responsible for 
convening the evaluation team and ensuring that others are included in all 
stages of the evaluation effort as appropriate. The team also should be 
responsible for acquiring outside expertise and assistance as needed. A larger 
team that includes community members can serve as an advisory group for 
the core team. 
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STAGE 1 . PLANNING THE EVALUATION 




D iscussing the details of an evaluation design (e.g., evaluation 

questions, measures, methods, analysis) are often where evaluators 
begin. This temptation should be resisted; take time to think and 
plan first! Considerable time and effort can be saved by completing four 
planning steps: 

• Define the purpose(s) of the evaluation. 

• Identify program evaluation requirements. 

® Understand the evaluation context. 

® Consider the use of an outside consultant. 

Define the Purp®se($) of the Ivoiyofion 

It is very important to be clear about what you expect your evaluation to 
accomplish. At a minimum, it should provide the data you need to meet the 
reporting requirements associated with maintaining funding for your new 
initiative. Equally important, however, is the evaluation of outcomes and 
impact and an analysis of the processes used to attain them in order to make 
programming adjustments along the way. The evaluation should examine 
progress, assess the accomplishment of intermediate objectives, and provide 
findings in a timely manner so that needed programmatic adjustments can be 
made as required. 

Identify Program E^aly®fi®n Requirements 

First among your planning tasks is the identification of all evaluation 
requirements that must be met. Requirements that must be considered in 
designing your comprehensive school reform evaluation are plentiful and 
come from a number of sources: 

® the federal government, 

0 state departments of education, 

® reform model developers, and 

° (of course) your district and/or building constituents. 
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Stage 1. Planning the Evaluation ° ° ° 

At one level, federal and state evaluation requirements are among the most 
important since continued funding, in part, depends on meeting them. States 
and districts participating in the CSRD Program were required to outline 
evaluation plans in their proposals. In many cases, states passed the 
responsibility for evaluation to funded districts. And don’t forget that many 
recommended CSRD models have evaluation components and requirements of 
their own. These requirements will vary depending on the model(s) you have 
chosen to incorporate into your reform program. (If you have adopted an 
established reform model, there will be a contact person with whom you can 
work. This guide identifies that person as the reform model developer.) Finally, if 
you are at the school level, there may well be evaluation questions about 
progress that require the collection of data detailed enough to guide teachers 
and administrators in making program adjustments. Add them to the mix! 

Federal guidelines. A review of the federal guidelines for CSRD Program 
evaluations 4 reveals the following requirements: 

• The primary focus of the evaluation is measuring program impact on 
student academic performance. But performance measures are not 
limited to student achievement. 

® Student performance measures should be clearly related to intended 
program outcomes. Both quantitative and qualitative data should be 
used. 

® Student achievement performance assessments should be the same ones 
used for accountability in Title I, and assessment results should be 
disaggregated by the categories specified in Title I to examine program 
impact on targeted student groups. 

® Performance measures should be compared with past performance at 
the same site, performance at similar sites within the district, or 
against national, state, or local student performance standards. 

0 Program implementation must be assessed. Both program start-up and 
long-term maintenance should be examined. 

° Areas that should be tracked for implementation include stakeholder 
support, parental participation, continuous staff development, and 
monitoring for performance. 

0 The nature, extent, source, and perceived usefulness of external 
technical assistance should be assessed. 

4 See Guidelines on the Comprehensive School Reform Demonstration Program. U.S. 
Department of Education, March 13, 1998, online at www.ed.gov/offices/OESE/ 
compreform/csrdgi .html . 
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Stage 1 . Planning the Evaluation 



These evaluation requirements are addressed as they relate to specific activities 
in the remaining stages of evaluation. In the planning stage, it is sufficient to 
simply identify, articulate, and align them. 

State and district requirements. State department of education solicitations 
for proposals are another source to examine to make sure that the evaluation 
you design meets specified guidelines. Also, your state probably has required 
funded districts and buildings to address evaluation in some specific way. 
Although you may feel that your proposal has responded completely to state 
requirements, it is worth another look. Continued funding may depend on 
compliance. As an example of what you might expect to find, Table 2 
presents the evaluation requirements from a CSRD proposal that a state 
submitted to the U.S. Department of Education. 



Toble 2 

ix®m|p0e State CS1P Ivsilyotoomi Heqyiiiremetnits 



The state will contract with an independent evaluator. 

Components to be evaluated will include the following: 

6 Leadership at the building level 
® Support from the district 
® Authenticity of implementation 

• Effect of professional development 

• Quality of parent involvement 

• Effect and quality of external support 

® Continuing commitment of faculty, staff, 
and administrators 

• Effect of CSRD Program on student achievement 

In addition, the district must design, describe in the application, 
and implement an evaluation of both the process and the results of 
the CSRD Program implemented in each school included in the 
application. Staff from each building must also describe how they 
plan to evaluate the program they are implementing. 



Reform model developer requirements. As part of the planning process, 
you should carefully examine what the developer of the reform model you 
have selected may expect of your building or district in relation to evaluation. 
You should note any data collection requirements and whether you will have 
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Using 

Worksheet 1 
to Align 
Requirements 



Using 

Worksheet 2 
to Organize 
Planning 
Information 



access to data or findings resulting from the developer’s work. If access is 
provided, consider using these data to answer evaluation questions, including 
those you have posed. 

Exhibit 1 illustrates how Worksheet 1 can be used to organize and align the 
various evaluation requirements that your CSRD Program must meet from 
federal and state agencies, your own district, and from the reform model 
developer. This exhibit shows how the sets of evaluation requirements overlap. 
Recognizing this redundancy at the outset can improve the efficiency and 
effectiveness of your evaluation design. 

Understand the Evaluation Context 

Next it is important to understand the local context in which the program 
will be implemented and the evaluation conducted. Contextual factors may 
affect your choice of evaluation design and data collection strategies and how 
key stakeholders use the results. Thus, the objective of this step in the process 
is to get the “lay of the land” and set the boundaries of the evaluation. 

Four areas should be considered: (1) local needs, (2) program elements, (3) 
stakeholder expectations, and (4) availability of evaluation resources and 
expertise. Worksheet 2 is designed to organize the information you develop as 
you respond to questions related to each of these four areas. Exhibit 2 offers a 
collection of possible responses to each of the questions. These examples are 
intended to stimulate thinking and to illustrate the types of responses you can 
expect to encounter. Overall, the worksheet is designed to help you organize 
the planning information that was collected through interviews and document 
review. 

Local needs . Your proposal is perhaps the best place to identify local needs or 
conditions requiring improvement. These may be explicitly stated or can be 
inferred from the proposal’s statement of goals and objectives. To gain a clear 
understanding of needs or conditions that can be improved, it may be 
necessary to expand your sources to include other school reports and the views 
of community members and school staff. You should attempt to answer the 
following question: 

® What local education needs or conditions were the impetus for 
developing the reform program? 

Program elements . When describing your reform program, you may want to 
include a synopsis of classroom activities, support services, or model 
components. Your proposal and the materials provided by your reform model 
developer are the best sources of information to describe your program. 
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1 . What local education needs or conditions were the impetus for developing the reform program? 



achievement gaps between ethnic minority and Anglo students' in reading 
and writing at ail grade, levels' 
low graduation rates 

Community ConCer n over poor math scores of 2th graders 



2. What are the major elements of the CSRD Program that are pertinent to the evaluation? 



alignment of curriculum and improved instruction in mat hematics 
pro f essional development to support diverse learning styles 
parent involvement /Community support 



3. What are the goals and objectives of the CSRD program? 



students proficient in math at 2th grade 
*5>0°/o increase in the time available to teachers for professional development 
increased parent involvement in Child's education 



4. Who is interested in your program and... 



what do they expect? 



district administrators 
teachers 
parents 

model developers 
media 



improved test scores 
more time for planning together 
Children s enthusiasm for school 
fidelity of model implementation 
better information about results 



5. What will be accepted as credible evidence of progress and impact by each individual or group? 



buildingwide adoption of district standards in sir months 
perceptions of teachers and parents , testimonials 
improved student performance on state assessments 



6. What resources and technical expertise are available to support the evaluation? 



/& days of tde. district e-vatuator per ye-ar 
2 dours of 3 te.acde.rs' time- per »vee/k 

funds for IO days of substitute, te.acde.rs' time, per ye.ar 



7 . What additional resources and expertise are needed? 
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Stage 1. Planning the Evaluation 



Nevertheless, upon review you may find that program descriptions in your 
proposal are not clear or succinct enough to meet present planning needs; 
therefore, interpretation may be necessary. Also, you may have to consider 
additions or changes made to your program since the proposal was written. 
You should attempt to answer the following questions: 

• What are the major elements of the CSRD Program that are pertinent 
to the evaluation? 

• What are the goals and objectives of the CSRD program? 

Stakeholders’ expectations . Stakeholders are those people who will be 
involved in the CSRD Program or who are concerned about its success. These 
individuals (or groups) will have much to say about your program and its 
results. Identify them early in the evaluation process, and involve them in the 
evaluation design process as much as possible. One way to begin identifying 
stakeholders is to learn who was involved in planning and writing your 
proposal. Identifying stakeholders and their expectations is only part of the 
task. You also will want to discern what will convince them that the program 
is successful. You should attempt to answer the following questions: 

® Who is interested in the program and what do they expect? 

• What will be accepted as credible evidence of progress and impact by 
each individual or group? 

Resources and expertise . Before designing your evaluation, it is important 
to determine the extent of resources — both financial and personnel — 
available to the evaluation effort. It is important to think beyond the design 
phase to the need for ongoing data collection, analysis, and reporting. Funds 
for evaluation may have been earmarked in your district’s proposal. 

What budget has been allocated for evaluation? This may be a single line 
item in the proposal budget or it may be distributed across a number of 
categories such as salaries, substitute teacher costs, materials development, 
copying, postage, printing, and consultants. It is also possible that the cost of 
conducting the evaluation has been completely overlooked in the budget. 

Estimating cost before a detailed design is in place is always difficult. 
However, experience shows that an adequate evaluation budget may be 
estimated at 10 percent of the total program cost. The cost will be somewhat 
less if it is possible to pool resources to develop designs and materials that can 
be used by multiple schools or districts. It is also possible that depending on 
the CSRD model selected, some provision for collecting and reporting 
evaluation information already has been made. It will be easier to estimate the 
cost of your evaluation needs after you have completed your design. A 
reassessment of your cost estimate at that time is advisable. 
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Finally, talk with those you expect to be members of the evaluation team to 
make sure they are aware of this assignment. Assess the evaluation skills and 
experience the team has in order to decide if using an outside consultant is 
indicated. The next section discusses the roles an outside consultant can play 
and how to go about selecting a qualified person. In reviewing the proposal 
and discussing evaluation expectations, you should attempt to answer the 
following questions: 

• What resources and technical expertise are available to support the 
evaluation? 

• What additional resources and expertise are needed? 

Consider Using a Consulfanf 

An outside consultant can assist at every stage of the evaluation. During the 
early phases, the consultant can help to frame the questions to be asked, 
identify or modify existing instruments, and negotiate with the reform model 
developer for access to planned data collection. Later on, the consultant can 
train staff in data collection methods, develop additional instruments such as 
checklists, and create systems for organizing and monitoring the evaluation 
process. Finally, the consultant can assist in analyzing and interpreting the 
results as well as framing multiple reports for different audiences. You should 
consider the role the consultant will play and the phases of the evaluation that 
you should use the consultant. 

There is no licensing or credentialing system for evaluators. Therefore, you 
should not hesitate to gather as much information about prospective evaluators 
as possible. In addition to the usual resume or vitae, be sure to obtain 
references directly from others who have worked with prospective consultants. 
Ask the evaluator for the names of former clients on projects that are similar to 
yours. Have the consultant submit a two- or three-page brief that describes 
how he or she is particularly qualified to assist you with your CSRD 
evaluation, what approach or method will be used in working with you, and 
what key issues will need to be discussed and resolved to complete a successful 
evaluation. Before making a selection, you should consider the adequacy and 
appropriateness of each prospective evaluator in view of their 

• formal preparation in evaluation, 

8 relevant experience, 

® philosophy or orientation to evaluation, and 

® personal style and characteristics. 
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Tips for Planning and Evaluation 



• Be sure to consider program improvement as well as demonstrating 
outcomes as purposes of your evaluation. 



• To get a feel for local conditions, review school reports and local 
newspaper articles; interview community members, parents, 
administrators, and teachers. 

• To learn about your CSR program, examine the proposal and talk with 
those who wrote it. 

• Find out who has an interest in the outcome of the program and what 
is expected. If differences exist between program goals and stakeholder 
expectations, stop planning and resolve them immediately. 

• Identify staff time, expertise, and resources allocated to complete the 
evaluation. 

• If resources are insufficient, consider working with nearby schools, 
districts, colleges, or education service agencies to reduce costs or 
acquire needed expertise. 

• Identify key time frames for evaluation information, for example, 
federal and state reporting cycles, community forums, and legislative 
sessions. 



• Start thinking now about how and by whom evaluation findings will 
be used. 
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STAGE 2. DESIGNING THE EVALUATION 

D esigning the evaluation involves specifying what will be included in 
the study and how it will be accomplished. Detailed guidance is 
provided in this section to help you design an evaluation of program 
context, implementation, and results. The recommendations included in this 
section are based on the assumption that the context and purpose of your 
program have been clarified and that an implementation strategy has been 
established. Only after these steps have been completed will it be possible to 
select appropriate measures of intermediate outcomes and impact to assess 
progress and appropriate measures of long-term outcomes and impact to assess 
the value of time and money invested in the reform effort. 

There are many steps involved in designing an evaluation, most of which are 
interrelated. What you learned and the determinations you made and 
documented on Worksheets 1 and 2 serve as a starting point for developing 
an evaluation design. We suggest that you also use this information to check 
the adequacy of your final evaluation design. Four major steps need to be 
completed in the design stage: 

• Link evaluation to the program. 

® Identify the evaluation questions. 

• Choose data sources, variables, and measures. 

• Move from planning to action. 

Link Evaluation to the Program 

In a comprehensive reform initiative, the program components are expected to 
work together toward the ultimate goal of improved student achievement. To 
make these connections, you should develop a program logic model that specifies 
the assumptions that link program components. Then review the logic of your 
program to identify gaps and inconsistencies. Remember that the outcomes 
are one focus of the evaluation; the others are implementation and the school 
or district context. For each of these areas of focus, you must develop 
evaluation questions to frame the design. (The process of developing these 
questions is described in an upcoming section.) 
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The program logic model (see Exhibit 3 on page 19) is a graphic 
representation of how CSRD Program activities are connected with desired 
outcomes or results. The logic model clarifies connections and underlying 
assumptions of your program and specifies intermediate outcomes. After you 
have completed your program logic model, you will be able to develop the 
program evaluation questions. Figure 1 illustrates how the program logic 
process leads to the identification of evaluation questions about context, 
implementation, and outcomes. 




Context 

Sssues 




Outcome 

Questions 



Context 

Questions 




Figure 1 * Identification of ivcaBucaifcin) ©yes? 



Using 

Worksheet 3 
to Create a 
Program Logic 
Model 



Exhibit 3 shows how Worksheet 3 is used to develop a logic model for your 
program that will drive the evaluation design and questions. To complete 
Worksheet 3, you will need your proposal, materials about implementation 
from the reform model developer, and a completed Worksheet 2. The first 
step in completing Worksheet 3 is to use the left-most box to list the local 
conditions, issues, or concerns that form the backdrop and that may influence 
the success of your reform effort. 



The next step is to use the Program Elements column to list the key aspects 
of the program. (It is not necessary to link specific program elements to the 
issues you listed in the left-hand box.) The program elements should not be 
tasks such as “schedule the professional development sessions with the reform 
model developer.” Rather, they should be fairly broad, for example 
“professional development” or “curriculum and assessment alignment.” If your 
proposal defines program components, these may serve the purpose. Some 
states are requiring schools to use the nine federal program requirements as 
I the components of their evaluation plans. 
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Stage 2. Designing the Evaluation ° ° ° 

Entries in the third column, Assumptions, along with entries in the fourth 
column, Intermediate Outcomes, are explicit statements of the underlying 
logic about how program elements lead to goals and outcomes. List the 
assumptions and intermediate outcomes underlying each program element in 
the boxes adjacent to the elements. You may want to write them as “if/then” 
statements. For example, if professional development is a program element, 
your statement might be “//teachers receive training in classroom 
management, then students will be on task more often.” 

Given the program elements you have specified, identify one or more 
assumptions and intermediate outcomes associated with each, as shown in 
Exhibit 3. If you need to establish more frequent benchmarks, you may have 
several “if’ and “then” statements in a row leading toward improved student 
achievement. However, because it will be necessary to collect data to 
determine if each outcome is being realized, it is best to limit the overall 
number of benchmarks to a well-selected set. Note that all of these outcomes 
are intended to move your school or district toward the ultimate goal of 
improved student achievement. 

Now review the logic model. Does your planned program make sense? If the 
intermediate outcomes are realized, will you accomplish your goals for 
improved student achievement? Are the assumptions sound? Are these the 
right set of activities to accomplish the intermediate goals? Are there gaps? 
There is little point in conducting an evaluation if the program is not well 
designed to accomplish its goals. 

Identify the Evaluation Questions 

Evaluation questions are the framework of the evaluation design. Exhibit 4 
includes examples of three kinds of evaluation questions — context, 
implementation, and outcomes — that should be recorded on Worksheet 4. 
Context questions address the factors that may influence your program and 
concerns related to the unique characteristics of your school (e.g., changes in 
population, changes in the economic make-up of the district). 5 Implementation 
questions are more practical. The overarching question is, “Did you do what 
you said you would do?” Outcomes questions ask, “Have you realized the 
intended goals or objectives?” or “What impact has the program had?” Draft 
a list of questions in each area. Then revisit your completed Worksheets 1 and 
2 to be certain you have included all the necessary areas to be addressed. 



Using 

Worksheet 4 
to Summarize 
Evaluation 
Questions 



5 Insight into the nature of contextual factors and implementation issues that can influence 
the outcome of reform efforts can be found in New American Schools after Six Years (1998) by 
T. K. Glennan, Jr. Santa Monica: Rand Education. 
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Choose Data Sources/ Variables/ and Measures 

One of the most important aspects of evaluation design is determining the 
best sources to use in answering the evaluation questions (as shown in Figure 
2). Documents, people, and events all are likely sources. The best sources are 
those that are closest to the matter you are evaluating, but using them may 
not be possible or cost effective. What you want to learn from each source is 
defined by the variables and measures. 



attitudes 

knowledge 
and skills 

aspiration 




capacities 

beliefs 

preparation 





Figure 2. Ev«s8u«sfii©n @ue$fli©rB$ ©rod tflhe Sources and Types of Information 
Needed to Answer Them. 



Using 

Worksheet 5 
to Create an 
Evaluation Design 
Matrix 



Exhibit 5 provides an example of a completed Worksheet 5. The evaluation 
design matrix is one of the primary components of the evaluation design. To 
complete Worksheet 5, begin by transferring the questions from Worksheet 4 
to the first column of Worksheet 5. Now you are ready to consider the best 
sources of data for each of the three kinds of questions. As you can see from 
the examples provided in Exhibit 5, sources of data include existing records 
and documents (e.g., student test scores, equipment inventory), people (e.g., 
teachers, students, parents), and events (e.g., meetings or classroom teaching 
sessions). Select the best source, or sources if needed, to provide data for each 
question. 



For each question and data source, you must also identify the relevant 
variables and measures. The variables operationalize — make specific — the 
evaluation questions. For example, as shown in Exhibit 5 for the context 
question, “Are all students benefiting from the reform?”, the variables are 
student attitudes and performance of students of different race and gender. 

As shown in Exhibit 5, measures are the specific means you will use to assign 
values to the variables. Using multiple measures of student performance is 
good practice. Supplement state assessments of student performance with locat- 
or school-developed assessments if they are available; these may be better 
aligned with local curricula. Include additional measures of student and school 
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Using 

Worksheet 6 
to Create an 
Evaluation 
Schedule 


performance in your evaluation to obtain a broader view of the value of your 
CSRD Program. Choose those that reasonably can be related to program goals. 
Examples include attendance, grade promotion, graduation, suspension, and 
expulsion rates; course-taking patterns; and parent time in classrooms. When 
you have completed the Variables and Measures columns, you should know 
exactly what information you will use to answer the evaluation questions. 

Move from Planning to Action 

With the questions and related data needs specified, you are now ready to 
specify what will be done, by whom, and when. This step involves defining 
meaningful tasks and realistic time lines. An important timing issue is when 
to collect baseline data, which are often needed to answer impact and outcome 
questions. In some cases (e.g., student test scores), data can be collected any 
time during the project; in other cases (e.g., teacher attitudes), data must be 
collected when the project begins in order to provide a basis for later 
comparisons. You should also review data reports that have already been 
completed to make use of existing data in your study. 

Other important sources of data include your reform model developer and the 
CSRD state evaluator. If the CSRD model you are using includes data 
collection, check the variables covered and time frames for collection. Find 
out how you can gain access to the data and whether you can add items as 
needed. In addition, check with your state coordinator to discuss state-level 
plans for CSRD evaluation. Again, data may be available through the work of 
a statewide evaluation and you may be able to add questions to a planned 
survey or interview. If these arrangements can be made, you will save 
considerable time not only for the data collectors but also in some cases for 
teachers, students, or parents who are the sources. You will also need to 
identify logistical issues that must be resolved (e.g., access to records, 
classrooms, and teachers), as well as specify areas where an outside consultant 
could be of assistance. In addition, virtually all of the external requirements 
(listed on Worksheet 1) for evaluating your CSRD Program must be 
addressed in this design stage. 

Exhibit 6 offers examples of how evaluation tasks can be organized into an 
evaluation schedule on Worksheet 6. To generate your list of evaluation tasks 
for the first column of Worksheet 6, begin by reviewing the evaluation 
questions. Next create a plan or schedule that organizes the work into tasks to 
be assigned to different members of the team. This will keep the questions 
central to your work and prevent you from losing sight of them as the work 
itself progresses. 
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By grouping evaluation questions that have common aspects, rather than 
working with them one by one, the evaluation design — and consequently 
data collection — can be made more efficient. One natural grouping is 
questions that rely on the same information source. For example, if there are 
several questions that can be answered by teachers, perhaps a single teacher 
survey is the most appropriate evaluation tool to efficiently collect the data 
needed to answer them. An alternative may be to group questions with a 
common focus. For instance, a context question about staff commitment may 
be conceptually connected to an implementation question about use of new 
materials. Thus, you may choose to collect data on commitment during 
interviews about use of materials. Of course, some questions will necessitate 
the use of more than one data collection method. 

In short, the intent of grouping questions into clusters is to help organize and 
streamline the data collection process. Still, your evaluation plan needs to be 
flexible, and this is one area you will want to refine as the evaluation 
proceeds. Defining and assigning the tasks as part of planning initiates the 
process. The next chapter provides guidance on selecting and implementing 
data collection methods. 



Tops for Designing on Evoiuotion 

• Focus initial evaluation efforts on implementation questions. The 
findings obtained can be used to improve program operations. 

• Don t forget to plan and schedule the collection of baseline data. 

• Plan to use existing data whenever possible and to integrate evaluation 
data collection into existing procedures as much as possible. 

• Coordinate the data collection and reporting requirements of your 
CSRD program with the requirements of other programs in place. For 
example, the reporting requirements of Title I clearly overlap with 
CSRD requirements. 

• Look for opportunities to use the same measures to answer multiple 
evaluation questions. 
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I t is time to get down to business and put the results of your planning and 
designing efforts into practice. Specifically, conducting the evaluation 
entails the following steps: 



• Obtain or create data collection instruments. 

• Collect the data. 

• Prepare the data for analysis. 

• Analyze the data. 



Obtain or Create Data Collection Instruments 

Data collection is the most time-intensive part of the evaluation. It is 
important to do all that you can to make it efficient. If staff members with 
the appropriate expertise are not available in-house, then consultants should 
be used to oversee instrument development, data collection, and data analysis. 



You should use available instruments whenever possible. The instruments you 
need will likely include surveys, checklists, observation tools, and interview 
protocols. Even if an existing instrument does not quite address all the data 
needs you have, it is easier to adapt it than to develop a new one from scratch. 
If you must develop your own instrument, be sure to pilot test it to identify 
unclear instructions and confusing items. This can be done with a sample of 
respondents from the same or parallel source (e.g., a few teachers from a 
different grade level than that needed for the evaluation). After you make 
necessary changes, the instrument is ready to use. 

Table 3 lists the advantages and disadvantages of common data collection 
procedures. Some concerns to keep in mind include protecting the 
confidentiality of respondents, considering time demands on both respondents 
and data collectors, and preparing those who will collect data. Interview 
guides and checklists are a fairly simple way to ensure consistent data 
collection across collectors and time periods. On the other hand, observations 
(e.g., of classroom teaching), make it more difficult to ensure data consistency 
and often require a data coding protocol and training of observers to be 
effective. 
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Procedure 


Advantages 


Disadvantages 


Self-admi nistered 
questionnaire 


Inexpensive. Can be quickly 
administered if distributed 
to group. Well suited for 
simple and short 
questionnaires. 


No control for 
misunderstood questions, 
missing data, or untruthful 
responses. Not suited for 
exploration of complex 
issues. 


Interviewer-administered 

questionnaires 


Interviewer can probe to 
ensure question is 
understood. With good 
rapport, may obtain useful 
open-ended comments, 
including evidence to 
support response. 


Confidentiality is an issue. 
May require hiring 
interviewers. Training is 
needed to establish 
consistency, nature, and use 
of probing questions. 


Open-ended interviews 


Usually yields richest data, 
details, new insights. Best if 
in-depth information is 
wanted. 


Same as above (interviewer- 
administered 

questionnaires); also often 
difficult to analyze. 


Focus groups 


Useful for gathering ideas 
and different viewpoints, 
discovering new insights, & 
improving question design. 


Not suited for 
generalizations about 
population being studied. 


Tests 


Provide ‘'hard” data, which 
administrators and funding 
agencies often prefer. 
Relatively easy to 
administer. Good 
instruments may be 
available from vendors. 


Available instruments may 
be unsuitable. Developing 
and validating new project- 
specific tests may be 
expensive and time 
consuming. Objections may 
be raised because of test 
unfairness or bias. 


Observations 


If well executed, best for 
obtaining data about 
behavior of individuals and 
groups. 


Usually expensive. Needs 
well-qualified staff. 
Observation may affect 
behavior being studied. 


Documents, records, and 
student work 


Existing materials can be 
used to develop data at a 
convenient time. 


Checklists or rubrics for 
generating data from the 
written material must be 
created. Careful definitions 
must be established to 
ensure consistency. 



Note: Adapted from User-Friendly Handbook for Project Evaluation (p. 44), by F. Stevens, F. Lawrenz, and 
L. Sharp (n.d.), Directorate for Education and Human Resources, National Science Foundation. 
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An important decision related to designing data collection instruments and 
strategies has to do with maintaining the confidentiality of information. 
Confidentiality helps respondents feel comfortable being honest because they 
know they will not be punished for their views. It also means that they can 
comment on another person’s performance without fear of damaging their 
relationship with that person. (If your design calls for a pre/post survey or test 
and you wish to maintain confidentiality, you will need to assign ID numbers 
to the respondents and keep a confidential file linking names with numbers 
for use at the time of the post-test.) Confidentiality also makes handling of 
the data more cumbersome, given the procedures necessary to protect identity. 
Anonymity is easier to handle — there is no identifying name or information 
on the instrument — but then you definitely cannot report the same kinds of 
information. And, you cannot easily remind those who have not yet 
completed the instrument that a response is needed. 

After you have selected or designed your instruments, it is important to 
reexamine them in view of the evaluation questions being addressed. Too 
often at this stage, the process and procedures of data collection take 
precedence over the substance of the questions to be answered. Be sure that all 
survey and interview items and observation protocols focus on the measures 
you have selected to answer each question. In addition, because you will in all 
likelihood be relying on existing databases or records (e.g., measures of 
student academic performance or student attendance), it will be necessary to 
develop a process for selecting the information you need and, in some cases, 
transferring it to a new database for use in the evaluation. 



Data collection involves scheduling, initial and follow-up information 
gathering, and data management. Timeliness in completing major data 
collection and analysis tasks and reporting the results to key stakeholders is 
most important. A data collection plan should specify when data will be 
collected, from whom it will be collected, and the steps needed to complete 
the process. 



On Worksheet 6 you prepared a broad overview of the evaluation tasks. Now 
refer to Exhibit 7, which offers an example of a completed data collection 
plan. To ensure consistent data collection across all sites and data collectors, 
the evaluation team should develop precise protocols, checklists, logs, and 
standardized forms. This task should be included in the data collection plan. 
Districts or schools should consider providing appropriate staff development 
opportunities to build staff members’ skills in data collection and analysis. 



Using 

Worksheet 7 
to Plan Data 
Collection 
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Tips for Conducting the Evaluation: 
Getting Started 



• To make ongoing data collection more efficient, create a list or 

database of contact information for various groups of individuals who 
will serve as sources of data. 



a consultant to start on the right foot if this expertise is not available 
. in-house. 

• Document data collection and data analysis procedures early in the 
evaluation process. You will need this information for reporting and 
for making revisions to the evaluation plan. 

Prepare the Data $@r Araalysis 

Data come in multiple forms. If you have designed your instruments well, 
you will have simplified the preparation steps. Quantitative data — from 
survey forms, checklists, or extracted from records — are generally ready to be 
entered into your database and used in analysis. 

Some checking for data coding problems is needed. How you will handle 
missing data and obviously incorrect data (e.g., a child’s age reported as 30), 
must be determined. In addition, some survey questions may have required a 
brief narrative response that must be converted into a form suitable for analysis. 

For example, perhaps you asked teachers to list their most frequently used 
teaching methods. In order to reduce the data to meaningful categories, you 
will need to compile a list of responses and then look for repeated methods. 
Each category should be assigned a unique identifier (e.g., a number or letter) 
so that you will be able to report the frequency of each of these methods. The 
person who creates the list must be able to determine when different wording 
still indicates the same method. 

Qualitative data require a more labor-intensive preparation and analysis 
process. (There are software programs you may want to use that can help you 
prepare and analyze data.^) For example, preparation of qualitative data will be 



^Renata Tesch (1990) discusses qualitative analysis programs in her book Qualitative 
Research: Analysis Types and Software Tools. 



Carefully plan data analysis procedures at the outset. Consider using 
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necessary if you conducted parent interviews and included open-ended 
questions about parents’ confidence in the school. The responses will likely 
vary in length and substance. Preparation of the data entail transcribing 
interviewer notes into a consistent format and then establishing a set of 
categories into which the responses can be organized and coded. 

Analyze the Data 

Too often, schools and districts gather evaluation data without having the 
appropriate methods for analyzing or interpreting the results. If you have kept 
your design reasonably simple, a first step is to report descriptive information. 
In the case of quantitative data, descriptive information typically includes 
frequencies of response (e.g., how many selected “strongly agree,” “agree,” 
“disagree,” “strongly disagree,” or “neutral”) and the average, or mean, 
response value. The results of this analysis could be reported for all 
respondents or reported for selected subgroups of respondents (e.g., teachers, 
administrators, parents). More complicated statistical analyses can be used to 
compare groups or subgroups or to note how people who responded to one set 
of questions responded to another related set of questions. 

At a simple level, analysis is the process of combining data from individuals 
into a common result. At a more meaningful level, it involves interpreting sets 
of findings and looking across them to better understand them and create a 
more comprehensive answer to an evaluation question. Interpreting data entails 
arriving at a reasonable explanation for the results obtained. To arrive at the 
best and most plausible interpretation of findings, it is important to first 
consider rival explanations and then to systematically rule out invalid options. 

To illustrate, if survey results reveal that parents are not comfortable with the 
reform process, a reasonable conclusion might be that change often is difficult 
and not supported by all stakeholders. However, plausible rival explanations 
might include that only disenchanted parents responded to the survey 
questions; this alternative explanation can be ruled out or validated by 
examining the process used for selecting parent respondents. If, in addition to 
parent survey responses about the reform, the findings from a focus group 
indicated that participating parents did not like to see things change, a 
stronger case for parent opposition to change can be made. In this case, the 
rival explanation that only a group of disenfranchised parents were 
uncomfortable with change can be more easily dismissed. 

The coded qualitative data also provide descriptive findings. You will be able 
to report the major themes that resulted from the data collection and whether 
groups differed in how they responded. Qualitative data often can make sense 
of discrepancies found in the quantitative data and validate or invalidate 
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alternative explanations. The analysis and interpretation tasks are the most 
important and most challenging of the evaluation process itself. 



validate them across all available sources. By presenting a draft of 
interpretations to various involved groups, each group’s comments and 
conclusions may further enrich the results. This process frequently engages 
stakeholders in developing recommendations for improvement. 



• Don’t rely solely on statistical analysis; other forms of analysis may 
better serve your needs. 

• Allow sufficient time to conduct thoughtful and in-depth analyses. 

• As a team, ask yourselves, “Do the results make sense? What are the 
possible explanations of findings and how will the results help us decide 
what actions will improve the program?” 

• Involve others in interpreting the results to gain insights from their 
experiences and to maintain their excitement about and involvement in 
the evaluation. 



The team needs to work together to develop explanations for the results and 
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N ow that the hard work is done, the really hard work begins. It is 

time to let those interested in the new program know just what you 
are learning about the program, its outcomes, and impact. Effective 
reporting of evaluation findings includes the following steps: 



• Organize study findings. 

• Select appropriate reporting media and formats. 

• Frame the findings as recommendations. 

• Deliver/disseminate the findings. 



Organize Study Findings 

Audiences for the evaluation findings may include district administrators and 
board members, parents and community members, the organization that 
developed and supports the reform model you have implemented, and state 
and school staff. On Worksheets 1 and 2, you already described the data needs 
and expectations of these various stakeholder groups. This information can be 
used to determine types of reports that you need to develop to best convey 
your findings and recommendations. Most audiences prefer a fairly succinct 
report; some may be interested in only one aspect of the information. 

Creating a master data file or report that compiles all background 
information, evaluation questions, analysis, results, findings, and 
recommendations provides the foundation for meeting various reporting 
needs. You might also choose to incorporate any evaluation reports completed 
by others (e.g., the state department of education, other schools in your 
district) in your file or report. Synthesizing your project evaluation in this 
way will help you meet the needs of the most demanding audience. 

Consider preparing shorter, successive evaluation reports that document the 
progress of implementation and immediate or interim outcomes. This will 
facilitate providing feedback to administrators and staff in a timely manner so 
they can make mid-course corrections as program implementation proceeds. It 
will also help others in the school district to make needed changes. 
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Developing interim reports of program results also will allow reform model 
developers to provide evidence to stakeholders of the benefits of the CSRD 
Program. This will help to generate “buy-in” from those stakeholders who 
may not have supported the CSRD Program initially. 

Select Reporting Media and Formats 

It is also important to ensure that your evaluation results are formatted in a 
way that stakeholders can easily use them. Reporting formats should be 
developed to link data to particular issues that concern government agencies, 
school boards, parents, and other stakeholders. For example, principals might 
prefer to receive computer-generated summaries of assessments, disaggregated 
by student groups receiving different types of instruction. School boards or 
state officials, on the other hand, might prefer to receive statistical progress 
reports with charts and graphs that compare student performance across years 
or buildings. Parents might prefer a narrative report that is formatted to 
respond to the key concerns they have already identified. In addition, be sure 
to develop reports in the languages that are relevant to your school community. 

Frame the Findings as Recommendations 

The CSRD evaluation is intended to be an integral part of the CSRD 
Program; it should be used to make program improvements. One of the most 
important ways that this can be realized is to frame the findings as specific 
recommendations for program improvement. If you engaged stakeholders in 
the interpretation process, recommendations may have already begun to 
emerge from that discussion. Therefore, suggestions for modification or 
change will not come as an uncomfortable surprise to these stakeholders. 

Nevertheless, putting the recommendations in writing will give them more 
weight and visibility with decision makers who have not been as directly 
involved in the evaluation process. For example, if there are aspects of the 
reform effort that are not working well, putting the recommendations for 
change in the evaluation report will encourage participants to make the 
changes than merely presenting a set of findings. Developing recommendations 
is neither an easy task nor one without consequence for the evaluation team — 
especially when there is substantial resistance to change. Working through the 
recommendations in advance with key decision makers is important. 



If you expect individuals or groups to take action, simply handing or mailing 
them a written report is seldom sufficient. When your findings are important 
for one or more groups to hear, consider presenting those findings orally and 
engaging each group in a discussion or question-and-answer session. Seek out 
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multiple occasions (perhaps as part of previously scheduled meetings) to keep 
your audiences informed. Placing brief articles in newsletters and posting 
information on available Web sites also are effective strategies for keeping 
stakeholders informed and interested. 




• Consider creating a “master” data file or report that compiles all the 
local context, program characteristics, baseline data, and evaluation 
findings to date. Once developed, the file is relatively easy to update 
with new data and findings. 

• Use the “master” data file to create reports and presentations tailored 
to key audiences. Consider developing at least the following: 

• a written executive summary of the findings; 

• an oral presentation supported with overheads or a computer- 
based presentation; and 

• a briefing paper that highlights key evaluation findings and 
recommendations. 

• Don’t forget to include recommendations for improvement in your 
reports and presentations. They are the blueprint for program 
improvement. 
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STAGE §« ENCOURAGING USE OF THE 

FINDINGS 



Y ou’re not quite done. Reporting the findings does not ensure that 

they will be used in any meaningful way. The value in the work you 
have done to evaluate your CSRD effort and understand the 
underlying conditions and causes of the progress made or the outcomes and 
impact realized rests in the opportunity to improve the effort with each new 
finding. Although your evaluation team may include a key administrator who 
can set directions based on findings, this is not usually sufficient to ensure 
that the findings will be integrated into the ongoing work of reform. 
Encouraging the use of the findings means enabling various stakeholders to 
more sharply focus their involvement and practices toward an even better 
result. In order to encourage stakeholders to use evaluation findings, consider 
the following: 

• Create opportunities to discuss findings. 

• Revise the evaluation plan as necessary. 

• Follow up with stakeholders to determine how findings are used. 



Create Opportunities to Discuss Findings 

Create opportunities to discuss evaluation findings and their use with 
administrators and others. Be sure to ask that sufficient time be made 
available to present your findings and to discuss recommendations. Keep your 
presentation short, but be specific about suggested changes. Allow time for 
open discussion with the teaching staff; you may even choose to present a 
draft of the report to the audience and then offer to incorporate their feedback 
in the final version. In addition, you may want to enlist students in reporting 
findings to parents, especially if they have had a role in data collection. 

Exhibit 8 illustrates how Worksheet 8 is used as a planning and tracking 
form to record your efforts to encourage the use of evaluation findings. 
Building a broad base of people who have heard the recommendations will 
support administrators in taking action. Therefore, be sure to plan to present 
the recommendations to a variety of audiences and key individuals. 



Using 

Worksheet 8 
to Summarize 
Actions 
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Revise Ev@lu@ifti@rD Ploou css Necessary 

Once you have determined how the findings were used and what other 
information might be helpful, you may need to revise your evaluation plan. 
This step of considering how the evaluation results can better reach the key 
players needs careful thinking. Of course, your plan will also need revising as 
the program is revised. 

Follow Up with Stakeholders 

You should follow up with at least the key members of your audiences to 
determine how they are using the findings and what other information needs 
they might have. On Worksheet 8 you can record actions taken as a result of 
your efforts to encourage use. If evaluation findings are not used in the early 
stages of the program, the reform may not be fully implemented. For 
example, if teachers lack needed materials and no attempt is made to supply 
them, they cannot possibly implement the program as planned. In later 
stages, as outcome results are available about what works and what does not, 
findings and recommendations should be incorporated into longer-range 
school planning. 



lops for Eracouragomg Use of the Findings 

® Be sure that findings are tailored for use by specific groups. 

• Be specific and give examples of what might be done in the future in 
your presentations and reports. 

• Let each stakeholder group know that using the findings is important 
and that you will be checking back to see if you can be of assistance. 



• Always follow up with key members of stakeholder groups to gain 
new insights. 
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WEB SITES 

http://www.mcrel.org/CSRD/ 

McREL’s CSRD Web site lists schools and districts in the region that are 
implementing comprehensive school reform models, including both funded and 
nonfunded sites. The Web site also provides descriptions of CSRD models and 
gives hyperlinks to reform model developers’ Web pages. In addition, it lists 
materials designed to help district and school leaders reallocate resources to 
support school reform. 
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http://www.ed.gov/offices/OESE/compreform/ 

The U.S. Department of Education’s CSRD Web site provides a variety of 
resources, including an extensive overview of the CSRD program, case studies of 
local efforts to implement CSRD models, and updates on CSRD program 
funding. 

http://www.ed.gov/inits/americareads/resourcekit/MakingInfo/miwfyl.html 

This page links to the U.S. Department of Education report Comprehensive 
Strategies for Children and Families: Collecting and Using Good Information for a Good 
Cause. Although not specifically related to CSRD, the report provides guidance 
on collecting and using data to design, implement, and evaluate comprehensive 
reform strategies. 

http://www.nwrd. org/ scpd/ natspec/catalog/ 

This page provides online access to the Catalog of School Reform Models (1st ed.) 
and its Addendum, developed by the nation’s regional educational laboratories to 
provide information on 26 entire-school, or whole-school, reform models and 18 
skill- and content-based models. 

http:www.temple.edu/LSS/CSR.htm 

This page provides a wide variety of resources, including a searchable database of 
state applications for federal CSRD funds, many of which contain descriptions of 
state requirements for local CSRD program evaluations. 

http://www.sedl.org/csrd/awards.html 

This page provides access to a searchable, nationwide database of schools that 
have been awarded CSRD funds. The database can be searched according to 
location, grade range (e.g., elementary, junior high/middle school, and 
senior/high school), and CSRD model. 

http://www.wcer.wisc.edu/ccvi/15_Regional_Centers_map.html 

This page gives contact information for the 1 5 Department of Education-funded 
Comprehensive Regional Assistance Centers. The mission of the centers is to 
provide technical assistance to Title I schoolwide programs and to help local 
education agencies (and schools funded by the Bureau of Indian Affairs) with 
high percentages or numbers of children in poverty. 
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WORKSHEETS FOR TEAM USE 

Worksheet 1: Alignment of Evaluation Requirements 

Worksheet 2: Evaluation Planning Questions 

Worksheet 3: Comprehensive School Reform Program Logic Model 

Worksheet 4: Summary of Evaluation Questions 

Worksheet 5: Evaluation Design Matrix 

Worksheet 6: Evaluation Design Schedule 

Worksheet 7 : Data Collection Plan 

Worksheet 8: Summary of Actions Taken to Encourage Use 

of Evaluation Findings 
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Worksheet 2: Evaluation Planning Questions 
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1 . What local education needs or conditions were the impetus for developing the reform 
program? 



2. What are the major elements of the CSRD Program that are pertinent to the evaluation? 



3. What are the goals and objectives of the CSRD program? 



4. Who is interested in your program and... 


what do they expect? 







5. What will be accepted as credible evidence of progress and impact by each individual or 
group? 



6. What resources and technical expertise are available to support the evaluation? 



7. What additional resources and expertise are needed? 
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Worksheet 8: Summary of Actions Taken to Encourage 
Use of Evaluation Findings 



Team Member 


Action Planned 


When 

Planned 


When 

Completed 


Outcome 
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